Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a virtual ShouldRetry method to the RetryPolicy for customization. #5584

Merged
merged 7 commits into from
May 14, 2024

Conversation

ahsonkhan
Copy link
Member

@ahsonkhan ahsonkhan commented Apr 29, 2024

Fixes #5515

Related discussion:
#5530 (comment)

Given this is not meant for end customers to modify, I did not add it to the changelog to avoid highlighting it.

cc @Jinming-Hu

Here is how I would expect the storage use case to be implemented. The StorageRetryPolicy would inherit from RetryPolicy, and override the ShouldRetry method:

  bool ShouldRetry(std::unique_ptr<RawResponse> const& response, RetryOptions const& retryOptions)
      const override
  {
    if (response == nullptr)
    {
      return true;
    }

    if (static_cast<std::underlying_type_t<Azure::Core::Http::HttpStatusCode>>(
            response->GetStatusCode())
        >= 400)
    {
      const auto& headers = response->GetHeaders();
      auto ite = headers.find("x-ms-copy-source-status-code");
      if (ite != headers.end())
      {
        const auto innerStatusCodeInt = std::stoi(ite->second);
        const auto innerStatusCode
            = static_cast<Azure::Core::Http::HttpStatusCode>(innerStatusCodeInt);

        const bool shouldRetry
            = retryOptions.StatusCodes.find(innerStatusCode) != retryOptions.StatusCodes.end();

        if (Azure::Core::Diagnostics::_internal::Log::ShouldWrite(
                Azure::Core::Diagnostics::Logger::Level::Informational))
        {
          Azure::Core::Diagnostics::_internal::Log::Write(
              Azure::Core::Diagnostics::Logger::Level::Informational,
              std::string("x-ms-copy-source-status-code ") + std::to_string(innerStatusCodeInt)
                  + (shouldRetry ? " will be retried" : " won't be retried"));
        }
        if (shouldRetry)
        {
          return true;
        }
      }
    }
    return false;
  }

Follow-up improvement to remove such tight coupling in existing tests: #5585

@ahsonkhan ahsonkhan self-assigned this Apr 29, 2024
@ahsonkhan ahsonkhan changed the title Add a virtual ShouldTry method to the RetryPolicy for customization. Add a virtual ShouldRetry method to the RetryPolicy for customization. Apr 29, 2024
@Jinming-Hu
Copy link
Member

@ahsonkhan can we merge this PR as soon as possible? This is blocking our STG94 work.

@RickWinter
Copy link
Member

@LarryOsterman can you please review as well.

@ahsonkhan
Copy link
Member Author

ahsonkhan commented May 7, 2024

can we merge this PR as soon as possible? This is blocking our STG94 work.

@Jinming-Hu We are waiting for this month's release cycle for May before merging this. So, I expect us to get this checked-in by eow. My current thinking is that we beta this as part of Core's release in June and GA it in July to align with when storage needs it.

@Jinming-Hu
Copy link
Member

@ahsonkhan core release is done. Can we merge this now?

@ahsonkhan ahsonkhan enabled auto-merge (squash) May 14, 2024 00:28
@ahsonkhan ahsonkhan merged commit ab90ef6 into Azure:main May 14, 2024
41 checks passed
@ahsonkhan ahsonkhan deleted the RetryExtension branch May 14, 2024 02:04
antkmsft added a commit to antkmsft/azure-sdk-for-cpp that referenced this pull request Jul 12, 2024
antkmsft added a commit that referenced this pull request Jul 12, 2024
* Revert "Update the RetryPolicy for the GA release, keeping ShouldRetry extension point hidden. (#5771)"

This reverts commit 9ccd206.

* Revert "Update the RetryPolicy and ShouldRetry customization logic to allow loosening the retry condition. (#5656)"

This reverts commit f1d9552.

* Do not remove changelog entry from a previous beta release

* Revert "Add a virtual ShouldRetry method to the RetryPolicy for customization. (#5584)"

This reverts commit ab90ef6.

---------

Co-authored-by: Anton Kolesnyk <[email protected]>
microzchang added a commit that referenced this pull request Jul 16, 2024
* Update vcpkg SHA (#5757)

Co-authored-by: Anton Kolesnyk <[email protected]>

* Sync eng/common directory with azure-sdk-tools for PR 8528 (#5755)

* Fix default value for env vars in build-test-resource-config template

* Add empty pool condition

---------

Co-authored-by: Ben Broderick Phillips <[email protected]>

* Sync eng/common directory with azure-sdk-tools for PR 8516 (#5752)

* Ensure subConfigFiles is not an empty string

* Skip instances where $file is an empty string

---------

Co-authored-by: Daniel Jurek <[email protected]>

* Sync eng/common directory with azure-sdk-tools for PR 8549 (#5762)

* commit the file changes so that we can see them running

* use standalone tool

---------

Co-authored-by: Scott Beddall (from Dev Box) <[email protected]>

* Disable codeql as it is causing build issues (#5761)

All the other languages have disabled codeql and we are now hitting blocking issues when running codeql on macos (https://dev.azure.com/twcdot/Data/_workitems/edit/138580) and even on the other OS's the configuration causes the builds to run for a long time.

* Delete eng/common/InterdependencyGraph.html which is unused. (#5763)

Co-authored-by: Wes Haggard <[email protected]>

* Add missing include (#5766)

* Update GitHubEventProcessor version to 1.0.0-dev.20240708.1 (#5768)

Co-authored-by: James Suplizio <[email protected]>

* Provide a walkthrough for how to install the Azure SDK packages in a VS (#5751)

MSBuild project via vcpkg.

* Sync eng/common directory with azure-sdk-tools for PR 8558 (#5764)

* Support storage network access and worm removal in remove test resources script

* Move storage network access script to common resource helpers file

* Improve storage container deletion resilience

* Plumb through pool variable to live test cleanup template

* Add sleep for network rule application

---------

Co-authored-by: Ben Broderick Phillips <[email protected]>

* add the ability to override default succeeded() conditioning by parameter (#5780)

Co-authored-by: Scott Beddall <[email protected]>

* Support regex/negative regex filters for stress test discovery. Add storage env defaults (#5779)

Co-authored-by: Ben Broderick Phillips <[email protected]>

* Update vcpkg SHA (#5772)

* Show-FailureLogs to include vcpkg build failure logs (#5776)

* Show-FailureLogs to include vcpkg build failure logs

* Add comment

* Add proper array syntax

* Use proper syntax to create an array even if there's only a single element

Co-authored-by: Ben Broderick Phillips <[email protected]>

---------

Co-authored-by: Anton Kolesnyk <[email protected]>
Co-authored-by: Ben Broderick Phillips <[email protected]>

* Turn federated auth on for Identity tests. (#5785)

* Turn federated auth on for Identity tests.

* Update test resources json.

* Move tests to use azure pipeline credentials  (#5754)

* test1

* hgdfchg

* remove the remnants of azure client secret

* test KV with federated auth

* UseFederatedAuth

* fdsa

* kv template with managed

* try try again

* retry permissions

* add net acls

* blunt force replace the resource json

* put back stuff

* trey again with new method

* attempt

* missed something

* flip if else

* Temporarily use empty sub config file path for preview cloud

* remove client secret

* try to fix the identity tests

* live skip failing tests and return in samples

* samples for identity fix

* disable failing samples in identity

* fix winhttp failing test

* comment out code

* remove managed identity

* restore version from main

* revert readme changes

* PR comments

* test 2

* clang

* attempt default creds with pipeline chanined

* clangs

* identity test and clangs

* oops

* live

* cleanup

* reter

* test

* revert the DAC change

* missed one

* taking the samples to a farm upstate

* PR comments

* Fix bad merge

---------

Co-authored-by: Daniel Jurek <[email protected]>
Co-authored-by: Anton Kolesnyk <[email protected]>
Co-authored-by: Anton Kolesnyk <[email protected]>

* Update the RetryPolicy for the GA release, keeping ShouldRetry extension point hidden. (#5771)

* Update the RetryPolicy for the GA release, keeping ShouldRetry extension
point hidden.

* Mark test helper virtual functions private, so they aren't
accessible/callable by callers.

* Update the changelog.

* Update CL.

* Azure Core July GA Release (#5792)

Co-authored-by: Anton Kolesnyk <[email protected]>

* Revert commits related to the new RetryPolicy method (#5793)

* Revert "Update the RetryPolicy for the GA release, keeping ShouldRetry extension point hidden. (#5771)"

This reverts commit 9ccd206.

* Revert "Update the RetryPolicy and ShouldRetry customization logic to allow loosening the retry condition. (#5656)"

This reverts commit f1d9552.

* Do not remove changelog entry from a previous beta release

* Revert "Add a virtual ShouldRetry method to the RetryPolicy for customization. (#5584)"

This reverts commit ab90ef6.

---------

Co-authored-by: Anton Kolesnyk <[email protected]>

* Sync eng/common directory with azure-sdk-tools for PR 8598 (#5777)

* Set storage account test resources to disable blob public access

* Skip adding network rules to storage accounts that don't need them during cleanup

* Add succeeded check to set pipeline subnet info step

* Disable network firewall by default in resource creation/removal

---------

Co-authored-by: Ben Broderick Phillips <[email protected]>

* Increment package version after release of azure-core (#5794)

* Acknowledge community contribution in the changelog (#5797)

* Mention community contribution in the changelog

* cspell

* Remove double spaces

---------

Co-authored-by: Anton Kolesnyk <[email protected]>

---------

Co-authored-by: Anton Kolesnyk <[email protected]>
Co-authored-by: Anton Kolesnyk <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Ben Broderick Phillips <[email protected]>
Co-authored-by: Daniel Jurek <[email protected]>
Co-authored-by: Scott Beddall (from Dev Box) <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: Robert Schulze <[email protected]>
Co-authored-by: James Suplizio <[email protected]>
Co-authored-by: Ahson Khan <[email protected]>
Co-authored-by: Ben Broderick Phillips <[email protected]>
Co-authored-by: George Arama <[email protected]>
microzchang added a commit to microzchang/azure-sdk-for-cpp that referenced this pull request Aug 7, 2024
* Update vcpkg SHA (Azure#5757)

Co-authored-by: Anton Kolesnyk <[email protected]>

* Sync eng/common directory with azure-sdk-tools for PR 8528 (Azure#5755)

* Fix default value for env vars in build-test-resource-config template

* Add empty pool condition

---------

Co-authored-by: Ben Broderick Phillips <[email protected]>

* Sync eng/common directory with azure-sdk-tools for PR 8516 (Azure#5752)

* Ensure subConfigFiles is not an empty string

* Skip instances where $file is an empty string

---------

Co-authored-by: Daniel Jurek <[email protected]>

* Sync eng/common directory with azure-sdk-tools for PR 8549 (Azure#5762)

* commit the file changes so that we can see them running

* use standalone tool

---------

Co-authored-by: Scott Beddall (from Dev Box) <[email protected]>

* Disable codeql as it is causing build issues (Azure#5761)

All the other languages have disabled codeql and we are now hitting blocking issues when running codeql on macos (https://dev.azure.com/twcdot/Data/_workitems/edit/138580) and even on the other OS's the configuration causes the builds to run for a long time.

* Delete eng/common/InterdependencyGraph.html which is unused. (Azure#5763)

Co-authored-by: Wes Haggard <[email protected]>

* Add missing include (Azure#5766)

* Update GitHubEventProcessor version to 1.0.0-dev.20240708.1 (Azure#5768)

Co-authored-by: James Suplizio <[email protected]>

* Provide a walkthrough for how to install the Azure SDK packages in a VS (Azure#5751)

MSBuild project via vcpkg.

* Sync eng/common directory with azure-sdk-tools for PR 8558 (Azure#5764)

* Support storage network access and worm removal in remove test resources script

* Move storage network access script to common resource helpers file

* Improve storage container deletion resilience

* Plumb through pool variable to live test cleanup template

* Add sleep for network rule application

---------

Co-authored-by: Ben Broderick Phillips <[email protected]>

* add the ability to override default succeeded() conditioning by parameter (Azure#5780)

Co-authored-by: Scott Beddall <[email protected]>

* Support regex/negative regex filters for stress test discovery. Add storage env defaults (Azure#5779)

Co-authored-by: Ben Broderick Phillips <[email protected]>

* Update vcpkg SHA (Azure#5772)

* Show-FailureLogs to include vcpkg build failure logs (Azure#5776)

* Show-FailureLogs to include vcpkg build failure logs

* Add comment

* Add proper array syntax

* Use proper syntax to create an array even if there's only a single element

Co-authored-by: Ben Broderick Phillips <[email protected]>

---------

Co-authored-by: Anton Kolesnyk <[email protected]>
Co-authored-by: Ben Broderick Phillips <[email protected]>

* Turn federated auth on for Identity tests. (Azure#5785)

* Turn federated auth on for Identity tests.

* Update test resources json.

* Move tests to use azure pipeline credentials  (Azure#5754)

* test1

* hgdfchg

* remove the remnants of azure client secret

* test KV with federated auth

* UseFederatedAuth

* fdsa

* kv template with managed

* try try again

* retry permissions

* add net acls

* blunt force replace the resource json

* put back stuff

* trey again with new method

* attempt

* missed something

* flip if else

* Temporarily use empty sub config file path for preview cloud

* remove client secret

* try to fix the identity tests

* live skip failing tests and return in samples

* samples for identity fix

* disable failing samples in identity

* fix winhttp failing test

* comment out code

* remove managed identity

* restore version from main

* revert readme changes

* PR comments

* test 2

* clang

* attempt default creds with pipeline chanined

* clangs

* identity test and clangs

* oops

* live

* cleanup

* reter

* test

* revert the DAC change

* missed one

* taking the samples to a farm upstate

* PR comments

* Fix bad merge

---------

Co-authored-by: Daniel Jurek <[email protected]>
Co-authored-by: Anton Kolesnyk <[email protected]>
Co-authored-by: Anton Kolesnyk <[email protected]>

* Update the RetryPolicy for the GA release, keeping ShouldRetry extension point hidden. (Azure#5771)

* Update the RetryPolicy for the GA release, keeping ShouldRetry extension
point hidden.

* Mark test helper virtual functions private, so they aren't
accessible/callable by callers.

* Update the changelog.

* Update CL.

* Azure Core July GA Release (Azure#5792)

Co-authored-by: Anton Kolesnyk <[email protected]>

* Revert commits related to the new RetryPolicy method (Azure#5793)

* Revert "Update the RetryPolicy for the GA release, keeping ShouldRetry extension point hidden. (Azure#5771)"

This reverts commit 9ccd206.

* Revert "Update the RetryPolicy and ShouldRetry customization logic to allow loosening the retry condition. (Azure#5656)"

This reverts commit f1d9552.

* Do not remove changelog entry from a previous beta release

* Revert "Add a virtual ShouldRetry method to the RetryPolicy for customization. (Azure#5584)"

This reverts commit ab90ef6.

---------

Co-authored-by: Anton Kolesnyk <[email protected]>

* Sync eng/common directory with azure-sdk-tools for PR 8598 (Azure#5777)

* Set storage account test resources to disable blob public access

* Skip adding network rules to storage accounts that don't need them during cleanup

* Add succeeded check to set pipeline subnet info step

* Disable network firewall by default in resource creation/removal

---------

Co-authored-by: Ben Broderick Phillips <[email protected]>

* Increment package version after release of azure-core (Azure#5794)

* Acknowledge community contribution in the changelog (Azure#5797)

* Mention community contribution in the changelog

* cspell

* Remove double spaces

---------

Co-authored-by: Anton Kolesnyk <[email protected]>

---------

Co-authored-by: Anton Kolesnyk <[email protected]>
Co-authored-by: Anton Kolesnyk <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Ben Broderick Phillips <[email protected]>
Co-authored-by: Daniel Jurek <[email protected]>
Co-authored-by: Scott Beddall (from Dev Box) <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: Robert Schulze <[email protected]>
Co-authored-by: James Suplizio <[email protected]>
Co-authored-by: Ahson Khan <[email protected]>
Co-authored-by: Ben Broderick Phillips <[email protected]>
Co-authored-by: George Arama <[email protected]>
microzchang added a commit that referenced this pull request Aug 7, 2024
* Storage/STG95 Files OAuth - full data plane support (#5759)

* File OAuth

* Remove comment

* update test record

* [Storage STG95 Branch] Merge Main to fix cmake build error (#5796)

* Update vcpkg SHA (#5757)

Co-authored-by: Anton Kolesnyk <[email protected]>

* Sync eng/common directory with azure-sdk-tools for PR 8528 (#5755)

* Fix default value for env vars in build-test-resource-config template

* Add empty pool condition

---------

Co-authored-by: Ben Broderick Phillips <[email protected]>

* Sync eng/common directory with azure-sdk-tools for PR 8516 (#5752)

* Ensure subConfigFiles is not an empty string

* Skip instances where $file is an empty string

---------

Co-authored-by: Daniel Jurek <[email protected]>

* Sync eng/common directory with azure-sdk-tools for PR 8549 (#5762)

* commit the file changes so that we can see them running

* use standalone tool

---------

Co-authored-by: Scott Beddall (from Dev Box) <[email protected]>

* Disable codeql as it is causing build issues (#5761)

All the other languages have disabled codeql and we are now hitting blocking issues when running codeql on macos (https://dev.azure.com/twcdot/Data/_workitems/edit/138580) and even on the other OS's the configuration causes the builds to run for a long time.

* Delete eng/common/InterdependencyGraph.html which is unused. (#5763)

Co-authored-by: Wes Haggard <[email protected]>

* Add missing include (#5766)

* Update GitHubEventProcessor version to 1.0.0-dev.20240708.1 (#5768)

Co-authored-by: James Suplizio <[email protected]>

* Provide a walkthrough for how to install the Azure SDK packages in a VS (#5751)

MSBuild project via vcpkg.

* Sync eng/common directory with azure-sdk-tools for PR 8558 (#5764)

* Support storage network access and worm removal in remove test resources script

* Move storage network access script to common resource helpers file

* Improve storage container deletion resilience

* Plumb through pool variable to live test cleanup template

* Add sleep for network rule application

---------

Co-authored-by: Ben Broderick Phillips <[email protected]>

* add the ability to override default succeeded() conditioning by parameter (#5780)

Co-authored-by: Scott Beddall <[email protected]>

* Support regex/negative regex filters for stress test discovery. Add storage env defaults (#5779)

Co-authored-by: Ben Broderick Phillips <[email protected]>

* Update vcpkg SHA (#5772)

* Show-FailureLogs to include vcpkg build failure logs (#5776)

* Show-FailureLogs to include vcpkg build failure logs

* Add comment

* Add proper array syntax

* Use proper syntax to create an array even if there's only a single element

Co-authored-by: Ben Broderick Phillips <[email protected]>

---------

Co-authored-by: Anton Kolesnyk <[email protected]>
Co-authored-by: Ben Broderick Phillips <[email protected]>

* Turn federated auth on for Identity tests. (#5785)

* Turn federated auth on for Identity tests.

* Update test resources json.

* Move tests to use azure pipeline credentials  (#5754)

* test1

* hgdfchg

* remove the remnants of azure client secret

* test KV with federated auth

* UseFederatedAuth

* fdsa

* kv template with managed

* try try again

* retry permissions

* add net acls

* blunt force replace the resource json

* put back stuff

* trey again with new method

* attempt

* missed something

* flip if else

* Temporarily use empty sub config file path for preview cloud

* remove client secret

* try to fix the identity tests

* live skip failing tests and return in samples

* samples for identity fix

* disable failing samples in identity

* fix winhttp failing test

* comment out code

* remove managed identity

* restore version from main

* revert readme changes

* PR comments

* test 2

* clang

* attempt default creds with pipeline chanined

* clangs

* identity test and clangs

* oops

* live

* cleanup

* reter

* test

* revert the DAC change

* missed one

* taking the samples to a farm upstate

* PR comments

* Fix bad merge

---------

Co-authored-by: Daniel Jurek <[email protected]>
Co-authored-by: Anton Kolesnyk <[email protected]>
Co-authored-by: Anton Kolesnyk <[email protected]>

* Update the RetryPolicy for the GA release, keeping ShouldRetry extension point hidden. (#5771)

* Update the RetryPolicy for the GA release, keeping ShouldRetry extension
point hidden.

* Mark test helper virtual functions private, so they aren't
accessible/callable by callers.

* Update the changelog.

* Update CL.

* Azure Core July GA Release (#5792)

Co-authored-by: Anton Kolesnyk <[email protected]>

* Revert commits related to the new RetryPolicy method (#5793)

* Revert "Update the RetryPolicy for the GA release, keeping ShouldRetry extension point hidden. (#5771)"

This reverts commit 9ccd206.

* Revert "Update the RetryPolicy and ShouldRetry customization logic to allow loosening the retry condition. (#5656)"

This reverts commit f1d9552.

* Do not remove changelog entry from a previous beta release

* Revert "Add a virtual ShouldRetry method to the RetryPolicy for customization. (#5584)"

This reverts commit ab90ef6.

---------

Co-authored-by: Anton Kolesnyk <[email protected]>

* Sync eng/common directory with azure-sdk-tools for PR 8598 (#5777)

* Set storage account test resources to disable blob public access

* Skip adding network rules to storage accounts that don't need them during cleanup

* Add succeeded check to set pipeline subnet info step

* Disable network firewall by default in resource creation/removal

---------

Co-authored-by: Ben Broderick Phillips <[email protected]>

* Increment package version after release of azure-core (#5794)

* Acknowledge community contribution in the changelog (#5797)

* Mention community contribution in the changelog

* cspell

* Remove double spaces

---------

Co-authored-by: Anton Kolesnyk <[email protected]>

---------

Co-authored-by: Anton Kolesnyk <[email protected]>
Co-authored-by: Anton Kolesnyk <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Ben Broderick Phillips <[email protected]>
Co-authored-by: Daniel Jurek <[email protected]>
Co-authored-by: Scott Beddall (from Dev Box) <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: Robert Schulze <[email protected]>
Co-authored-by: James Suplizio <[email protected]>
Co-authored-by: Ahson Khan <[email protected]>
Co-authored-by: Ben Broderick Phillips <[email protected]>
Co-authored-by: George Arama <[email protected]>

* Storage/STG95 Premium Files Paid Burst (#5795)

* paid burst

* update record

* Storeage/STG95 Files Binary ACE (#5824)

* feature code

* update test case to playback only

* update records

* fix cspell

* update record

* fix cspell

* fix cspell

* fix comments

* Remove copy opreation rest code.

* Storage/STG95 Support Sas StringToSign for debugging (#5830)

* StringToSign

* Update deprecate

* Update Func name

* ignore deprecated warning

* Add missing ignore

* fix clang format

* Storage/STG95 Resolve Archboard Review Comments (#5856)

* fix comments

* fix comments

* Add Create dir and upload from support (#5866)

* merge test recordings

* Remove other changes

* Remove deprecated (#5873)

* Remove others

---------

Co-authored-by: Anton Kolesnyk <[email protected]>
Co-authored-by: Anton Kolesnyk <[email protected]>
Co-authored-by: Azure SDK Bot <[email protected]>
Co-authored-by: Ben Broderick Phillips <[email protected]>
Co-authored-by: Daniel Jurek <[email protected]>
Co-authored-by: Scott Beddall (from Dev Box) <[email protected]>
Co-authored-by: Wes Haggard <[email protected]>
Co-authored-by: Robert Schulze <[email protected]>
Co-authored-by: James Suplizio <[email protected]>
Co-authored-by: Ahson Khan <[email protected]>
Co-authored-by: Ben Broderick Phillips <[email protected]>
Co-authored-by: George Arama <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Retry policy supports service-specific retry logic
4 participants